Calibration#
The Calibration class provides a way to adjust weights of observations in a dataset to match specified target values. This is commonly used in survey research and policy modeling for rebalancing datasets to better represent desired population characteristics.
The calibration process uses an optimization algorithm to find weights that minimize the distance from the original weights while achieving the target constraints.
Basic usage#
Parameters#
__init__(data, weights, targets)
data(pd.DataFrame): The dataset to be calibrated. This should contain all the variables you want to use for calibration.weights(np.ndarray): Initial weights for each observation in the dataset. Typically starts as an array of ones for equal weighting.targets(np.ndarray): Target values that the calibration process should achieve. These correspond to the desired weighted sums.
Calibration can be easily done by initializing the Calibration class, passing in the parameters above. Then calibrate() method performs the actual calibration using the reweight function. This method:
Adjusts the weights to better match the target values
Updates both
self.weightsandself.datawith the calibrated results
Example#
Below is a complete example showing how to calibrate a dataset to match income targets for specific age groups:
from microcalibrate.calibration import Calibration
import logging
import numpy as np
import pandas as pd
import plotly.graph_objs as go
from plotly.subplots import make_subplots
logging.basicConfig(
level=logging.INFO,
)
# Create a sample dataset with age and income data
random_generator = np.random.default_rng(0)
data = pd.DataFrame({
"age": np.append(random_generator.integers(18, 70, size=120), 71),
"income": random_generator.normal(40000, 10000, size=121),
})
# Set initial weights (all one in this example)
weights = np.ones(len(data))
# Calculate target values: total income for age groups 20-30 and 40-50 (as an example) or employ existing targets
targets_matrix = pd.DataFrame({
"income_aged_20_30": ((data["age"] >= 20) & (data["age"] <= 30)).astype(float) * data["income"],
"income_aged_40_50": ((data["age"] >= 40) & (data["age"] <= 50)).astype(float) * data["income"],
"income_aged_71" : (data["age"] == 71).astype(float) * data["income"],
})
# 15% higher than the sum of data with the original weights
targets = np.array([
(targets_matrix["income_aged_20_30"] * weights * 1000).sum(),
(targets_matrix["income_aged_40_50"] * weights * 1.15).sum(),
(targets_matrix["income_aged_71"] * weights * 1.15).sum()
])
print(f"Original weights: {weights}")
print(f"Original targets: {targets}")
Original weights: [1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
1.]
Original targets: [7.37032429e+08 9.76779350e+05 4.36479914e+04]
# Initialize the Calibration object
calibrator = Calibration(
estimate_matrix=targets_matrix,
weights=weights,
targets=targets,
noise_level=0.05,
epochs=528,
learning_rate=0.01,
dropout_rate=0,
)
# Perform the calibration
performance_df = calibrator.calibrate()
print(f"Original dataset size: {len(targets_matrix)}")
print(f"Calibrated dataset size: {len(calibrator.estimate_matrix)}")
print(f"Number of calibrated weights: {len(calibrator.weights)}")
INFO:microcalibrate.calibration:Performing basic target assessment...
WARNING:microcalibrate.calibration:Target income_aged_20_30 (7.37e+08) differs from initial estimate (7.37e+05) by 3.00 orders of magnitude.
WARNING:microcalibrate.calibration:Target income_aged_71 is supported by only 0.83% of records in the loss matrix. This may make calibration unstable or ineffective.
INFO:microcalibrate.reweight:Starting calibration process for targets ['income_aged_20_30' 'income_aged_40_50' 'income_aged_71']: [7.37032429e+08 9.76779350e+05 4.36479914e+04]
INFO:microcalibrate.reweight:Original weights - mean: 1.0000, std: 0.0000
INFO:microcalibrate.reweight:Initial weights after noise - mean: 1.0245, std: 0.0143
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.339, weights_mean=1.02, weights_std=0.0143, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 33.33%
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.333, weights_mean=1.06, weights_std=0.0542, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.333, weights_mean=1.09, weights_std=0.0952, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.1, weights_std=0.135, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.12, weights_std=0.188, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.15, weights_std=0.254, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 52: Loss = 0.332152, Change = 0.006785 (improving)
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.19, weights_std=0.331, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.23, weights_std=0.423, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.332, weights_mean=1.27, weights_std=0.534, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.331, weights_mean=1.33, weights_std=0.666, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.331, weights_mean=1.39, weights_std=0.826, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 104: Loss = 0.330964, Change = 0.001187 (improving)
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.331, weights_mean=1.47, weights_std=1.02, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.33, weights_mean=1.57, weights_std=1.26, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.33, weights_mean=1.7, weights_std=1.55, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.329, weights_mean=1.85, weights_std=1.92, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.328, weights_mean=2.04, weights_std=2.37, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 156: Loss = 0.327622, Change = 0.003342 (improving)
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.327, weights_mean=2.28, weights_std=2.95, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.326, weights_mean=2.58, weights_std=3.68, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.324, weights_mean=2.97, weights_std=4.61, weights_min=1]
Reweighting progress: 0%| | 0/528 [00:00<?, ?epoch/s, loss=0.322, weights_mean=3.47, weights_std=5.81, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.322, weights_mean=3.47, weights_std=5.81, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.319, weights_mean=4.11, weights_std=7.36, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 208: Loss = 0.316212, Change = 0.011410 (improving)
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.315, weights_mean=4.95, weights_std=9.38, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.311, weights_mean=6.06, weights_std=12, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.304, weights_mean=7.51, weights_std=15.5, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.296, weights_mean=9.44, weights_std=20.1, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.285, weights_mean=12, weights_std=26.3, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.271, weights_mean=15.4, weights_std=34.5, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 260: Loss = 0.271425, Change = 0.044787 (improving)
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.253, weights_mean=20, weights_std=45.6, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.23, weights_mean=26.2, weights_std=60.4, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.2, weights_mean=34.4, weights_std=80.2, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.164, weights_mean=45.4, weights_std=107, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.121, weights_mean=59.9, weights_std=141, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 66.67%
INFO:microcalibrate.reweight:Epoch 312: Loss = 0.112365, Change = 0.159060 (improving)
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.0762, weights_mean=78.6, weights_std=186, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.0353, weights_mean=101, weights_std=241, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.00842, weights_mean=126, weights_std=300, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.00012, weights_mean=147, weights_std=350, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.000695, weights_mean=157, weights_std=374, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 100.00%
INFO:microcalibrate.reweight:Epoch 364: Loss = 0.000801, Change = 0.111564 (improving)
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=0.000513, weights_mean=156, weights_std=371, weights_min=1]
Reweighting progress: 37%|███▋ | 194/528 [00:00<00:00, 1936.49epoch/s, loss=4.46e-5, weights_mean=151, weights_std=361, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=4.46e-5, weights_mean=151, weights_std=361, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=9.52e-6, weights_mean=149, weights_std=355, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=2.1e-5, weights_mean=149, weights_std=354, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=4.93e-6, weights_mean=149, weights_std=356, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 100.00%
INFO:microcalibrate.reweight:Epoch 416: Loss = 0.000000, Change = 0.000801 (improving)
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=2.06e-9, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=6.66e-7, weights_mean=150, weights_std=358, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=2.96e-7, weights_mean=150, weights_std=358, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=6.69e-9, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=1.86e-8, weights_mean=150, weights_std=357, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 100.00%
INFO:microcalibrate.reweight:Epoch 468: Loss = 0.000000, Change = 0.000000 (improving)
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=1.41e-8, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=8.96e-10, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=4.98e-10, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=6.16e-10, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=5.51e-11, weights_mean=150, weights_std=357, weights_min=1]
Reweighting progress: 74%|███████▎ | 389/528 [00:00<00:00, 1940.47epoch/s, loss=1.65e-11, weights_mean=150, weights_std=357, weights_min=1]
INFO:microcalibrate.reweight:Within 10% from targets: 100.00%
INFO:microcalibrate.reweight:Epoch 520: Loss = 0.000000, Change = 0.000000 (improving)
Reweighting progress: 100%|██████████| 528/528 [00:00<00:00, 1941.00epoch/s, loss=1.65e-11, weights_mean=150, weights_std=357, weights_min=1]
INFO:microcalibrate.reweight:Reweighting completed. Final sample size: 121
Original dataset size: 121
Calibrated dataset size: 121
Number of calibrated weights: 121
# Calculate final weighted totals
final_totals = targets_matrix.mul(calibrator.weights, axis=0).sum().values
print(f"Target totals: {targets}")
print(f"Final calibrated totals: {final_totals}")
print(f"Difference: {final_totals - targets}")
print(f"Relative error: {(final_totals - targets) / targets * 100}")
Target totals: [7.37032429e+08 9.76779350e+05 4.36479914e+04]
Final calibrated totals: [7.37025455e+08 9.76778403e+05 4.36469951e+04]
Difference: [-6.97363547e+03 -9.46902634e-01 -9.96308503e-01]
Relative error: [-9.46177563e-04 -9.69413035e-05 -2.28259874e-03]
np.testing.assert_allclose(
final_totals,
targets,
rtol=0.01, # relative tolerance
err_msg="Calibrated totals do not match target values",
)
performance_df.head()
| epoch | loss | target_name | target | estimate | error | abs_error | rel_abs_error | |
|---|---|---|---|---|---|---|---|---|
| 0 | 0 | 0.338936 | income_aged_20_30 | 7.370324e+08 | 7.573207e+05 | -7.362751e+08 | 7.362751e+08 | 0.998972 |
| 1 | 0 | 0.338936 | income_aged_40_50 | 9.767794e+05 | 8.729661e+05 | -1.038132e+05 | 1.038132e+05 | 0.106281 |
| 2 | 0 | 0.338936 | income_aged_71 | 4.364799e+04 | 3.984997e+04 | -3.798020e+03 | 3.798020e+03 | 0.087015 |
| 3 | 52 | 0.332152 | income_aged_20_30 | 7.370324e+08 | 1.316427e+06 | -7.357160e+08 | 7.357160e+08 | 0.998214 |
| 4 | 52 | 0.332152 | income_aged_40_50 | 9.767794e+05 | 9.776910e+05 | 9.116250e+02 | 9.116250e+02 | 0.000933 |
g20 = performance_df.query("target_name == 'income_aged_20_30'")
g40 = performance_df.query("target_name == 'income_aged_40_50'")
fig = make_subplots(
rows=2, cols=2,
subplot_titles=[
"Estimate vs target: income_aged_20_30",
"Estimate vs target: income_aged_40_50",
"Relative absolute error: income_aged_20_30",
"Relative absolute error: income_aged_40_50",
],
shared_xaxes=True,
vertical_spacing=0.12,
horizontal_spacing=0.10,
)
fig.add_trace(
go.Scatter(
x=g20["epoch"], y=g20["target"],
mode="lines", name="Target 20-30",
line=dict(dash="dot", color="red"),
),
row=1, col=1,
)
fig.add_trace(
go.Scatter(
x=g20["epoch"], y=g20["estimate"],
mode="lines", name="Estimate 20-30",
line=dict(color="blue"),
),
row=1, col=1,
)
fig.add_trace(
go.Scatter(
x=g40["epoch"], y=g40["target"],
mode="lines", name="Target 40-50",
line=dict(dash="dot", color="red"),
),
row=1, col=2,
)
fig.add_trace(
go.Scatter(
x=g40["epoch"], y=g40["estimate"],
mode="lines", name="Estimate 40-50",
line=dict(color="green"),
),
row=1, col=2,
)
fig.add_trace(
go.Scatter(
x=g20["epoch"], y=g20["rel_abs_error"],
mode="lines", showlegend=False,
line=dict(color="blue"),
),
row=2, col=1,
)
fig.add_trace(
go.Scatter(
x=g40["epoch"], y=g40["rel_abs_error"],
mode="lines", showlegend=False,
line=dict(color="green"),
),
row=2, col=2,
)
fig.update_layout(
height=800, width=1050,
title_text="Calibration performance over epochs",
legend=dict(x=1.05, y=1, xanchor="left", yanchor="top"),
margin=dict(r=200),
)
fig.update_xaxes(title_text="Epoch", row=2, col=1)
fig.update_xaxes(title_text="Epoch", row=2, col=2)
fig.update_yaxes(title_text="Income ($)", row=1, col=1)
fig.update_yaxes(title_text="Income ($)", row=1, col=2)
fig.update_yaxes(title_text="Relative absolute error", row=2, col=1)
fig.update_yaxes(title_text="Relative absolute error", row=2, col=2)
fig.show()
summary = calibrator.summary()
summary
| Metric | Official target | Final estimate | Relative error | |
|---|---|---|---|---|
| 0 | income_aged_20_30 | 7.370324e+08 | 7.370273e+08 | -0.000007 |
| 1 | income_aged_40_50 | 9.767794e+05 | 9.767784e+05 | -0.000001 |
| 2 | income_aged_71 | 4.364799e+04 | 4.364699e+04 | -0.000023 |